Two words stand to change the face of computer gaming as we know it: Distributed Simulation.
There are a lot of emergent technologies which will have big impacts on the gaming industry as a whole—VR/Augmented Reality, Cloud Gaming (PC Gamer), Deviceless Interfaces, and Immersion Haptics to name a few—but Distributed Simulation dwarfs the stand-alone impact of other technologies because it is a force multiplier. It enhances the state of what they can do.
For those who need a tech primer
Distributed computing is where a computational load or processing effort is shared across a network of computers – each with their own processors and memory – to reduce the time or spend required to complete a task. In essence, distributed computing lets you do with a dozen or more $1,000 computers what you would otherwise need a high-end server or supercomputer for, at a fraction of the price. As the old adage goes, “Many hands make light work.”
Distributed Simulation is the love-child of distributed computing and simulation design, and it’s been around since at least the 1990’s. It started in Defense circles as quietly developed strategic programmes such as SIMNET and Chaos, for simulating and modelling engagements and threat scenarios like nuclear deployments. Later, as it entered the ‘gray market’ (that space occupied by private corporate developments) the technology matured as companies sought to develop ways to amplify their work without relying on big mainframes to do the heavy lifting.
Many prominent technical universities and colleges offer courses on distributed simulation, including Georgia Tech. However, it has only really been since 2005 that consumer gaming has entered the scope of considered applications in earnest. One game developer has commented that distributed physics “allows us to scale to an almost infinite number of physically simulated bodies without degrading physics performance.”
Why EVE?
Hadean openly state that this test is to measure the impact of running at large-scale in a single environment. Most MMOs conquer this by constraining player numbers.
For example, World of Warcraft deals with their incredible user counts (10M+ players) by having multiple dozens of servers per region. Then they have the users in those servers divided into different play zones as they move through the landscape. This segments how much user engagement goes on in the same place at the same time. WoW is still dealing with huge user numbers, but they have a lot of expensive server hardware. I don’t want to guess at their infrastructure spend.
EVE Online, however, puts everyone in one ‘shard’. All the players live in one universe on one server environment. That effort is divided over multiple servers, and there are beefier nodes for big fights, but it’s all one simulation. When enough players move into the same system, on the same grid, the only reason the entire system doesn’t come crashing down is the years of experience and the continual uplift projects CCP runs in their environment. Clever code and good hardware.
What Hadean has planned with their Aether Engine running on HadeanOS—an Operating System based on the Linux kernel but significantly different from it (tech details linked)—is to provide a platform which can make EVE Online’s biggest records look like a kindergarten play date. Even this conservative first load test is earmarked at 10,000 active players. The goal is to test peak user load and user counts, and unplanned issues in load-balancing (what they call ’emergent behaviour’, literally the ‘whoops’ factor). They also want to identify areas where there’s room for optimization, or where the system, network, or client are outright bleeding memory like a stuck pig.
Potential For Problems
The horror of a test like this is the hundreds of thousands of things which can go wrong. Over 80% of that isn’t even on Hadean’s end. Client disconnects can happen to people with unreliable internet. Participants may experience unfavorable conditions if their computers don’t meet the requirements of the test. Queuing issues could arise as people join the shard, or local processing issues if their computers can’t keep up with Hadean’s distributed simulation model, even though Hadean’s side is doing most of the heavy lifting.
Hadean have partnered with Microsoft for this 10K player deathmatch. The test is running on Azure, so some potential problems might be on Microsoft. They have also unleashed their own Netflix-inspired chaos monkey(2) to test their environment. Azure player bots run in different regions, in different machines. They even go as far as listing the hardware they’re testing on and showing examples of early tests. They are taking this seriously.
This kind of test also has a social impact. EVE players are a good choice for it. We’re a concentrated community; small player base and relatively competent players. However, with that comes the nature of our community. We’re unkind to shows of weakness and unforgiving of failures. You would be hard pressed to find a more critical bunch.
Yeah, Hadean’s got their hands full. They’re taking a big, bold risk running a huge test this publicly.
While not openly stated or confirmed, it is likely that Hadean’s other goals are the investor confidence and public esteem which even running this test will buy them. If they can pull it off it will mean investment dollars, market segment, and a lot of open doors. But is this a bold move toward something which can change the gaming industry, or a publicity stunt?
Quantum Leap, or Jumping the Shark?
CCP Hellmar has thrown his name behind this exercise, and CCP has sewn their brand to the Aether Engine test in more than one way. It will be interesting to see what the windfall or fallout will be from this move.
The question that stands at the fore is, “Is this thing gonna work?”
The second question is, “What does this mean for the future of gaming?”
In his dev blog, CCP Hellmar wrote:
“The partnership with Hadean is about more drastic, fundamental changes and is currently in the domain of cutting-edge tech research and development. Given that it’s still in its early stages, it’s not possible to put any timeframe on at this point, or even if the various attempts will work out the way we want them to. I do believe though, that this type of activities will be huge steps in the process of making sure that EVE really is forever.”
I see where he’s going with this, but what he’s said is sales-talk. Nothing Hadean is doing is truly novel. Distributed simulation is not a novel idea, nor is relying on the cloud to accomplish it. Other than the HadeanOS and Aether Engine, there’s nothing new being utilized to implement this test. I would be more surprised to see if there was a D-Wave quantum computer being utilized for load balancing, or AI-driven algorithms handing Quality of Service (QoS) and request prioritization.
I am not sure if what Hadean is doing qualifies as a quantum leap. If there is some secret sauce other than the attention and intention they’re giving applying distributed simulation to gaming, they haven’t said what it is. There are economies of scale at play here which have the potential to rain quite heavily on their proverbial parade.
Down the Road
This is a good first test, but even if it is a stellar success their OS may not survive the transition to scale at the level of the AAA studios like Activision, EA, and Bungie. Microsoft will likely find a way to use their platform for their game holdings like The Coalition, or providing back-end muscle to applications of the Hololens 2. Maybe Google or Microsoft move to buy Hadean by the end of this year.
Further along, personal computers (and mobile devices) have been growing in capacity and power. Computer languages are growing in efficiency. The need for all this extra compute is quickly losing ground. There will come a day when leveraging Azure at all may not be necessary, though at this point many may argue I’m being a Utopian idealist. But what Hadean is doing runs the very real risk of becoming commonplace in very short order. And there are others with the ability to take what they’re doing and do it better.
However, I agree with CCP Hellmar on one point: EVE needs the ability to pivot into a bigger, distributed architecture. It needs to move toward a model where EVE can ‘be forever’—or at least as long as people keep paying for it. I believe, though it hasn’t been said, that the not-so-long-term goal of CCP is to move away from the monolith. If this test is successful, it opens the door for that to happen. A lot of the heavy lifting from the client-side environment can be offloaded as well, and that opens doors for other new and novel development at CCP.
This month, we may find out what the shape of the future will be.
Afterthought: The Hadean Horror
P.S.: I think that it is very interesting that CCP released the Hadean Horror skin set for Abyssal ships, and even named the ‘Abyss’ expansion, not all that long before a partnership with Hadean was released. The word ‘Hadean’ comes from Greek Mythology, meaning ‘from Hades’. CCP is famous for using names from mythology for ships, modules, expansions, and the like so it could be a coincidence.
